⛶
The jury test#
16 Oct 2026

Work in progress. Which steps an analysis may be built from if somebody else is to rebuild and check it, and how far to break it down. Part of Pilots, not passengers — reconstructible reporting in evaluation. For one published set of steps see the open format; for the principles, the principles page.

If somebody else is to rebuild an analysis and check it, the analyst has to set out the working as steps. The analyst may be a person, a person working with AI, or an AI on its own. Rubicon, for example, lets the model reason however it likes, but will not accept the answer until the model has set out its working as a tree of steps someone else could check. Which steps, though? What may the working be built from? For a while we did not have a good answer.

As a question of logic, it has none. You can always invent one more operation, and a clever enough analysis will want one. A reasoning step could be almost anything, so a list of basic steps can never be proved complete.

A better question is which operations a person can follow. That question has an answer, because the person at the other end does not change with the subject. An untrained reader can follow only a few operations without being taught anything first, and these are much the same whatever the analysis is about: count, compare, group, filter, total, weigh one thing against another, follow a chain one link at a time. The list describes readers, so a new field does not lengthen it.

Juries have worked this way for centuries. A complicated case is broken down and explained to twelve people who know nothing about the field. They are not taught the discipline first. They are expected to follow the case anyway and reach a verdict. If a case cannot be put to them in steps they can follow, other people cannot test it at all.

So the jury does two jobs. It limits the set of steps, because only what a lay reader can follow is allowed in. It also tells us what we are after. In this setting we take rigour to mean that a claim is justified when a competent person who was not there can follow the reasoning and decide for themselves whether to accept each step. However confident an analysis is, we do not count it as justified if it cannot be reduced to such steps.

The same test settles how fine the steps should be. We want the coarsest step a lay reader can still check and accept. Logic would allow far finer steps, more than any reader would get through. A coarser step can contain a leap that the reader never sees, such as an unstated judgement.

We do not mean a literal jury, picked at random. The reader we have in mind is a smaller group with slightly more to go on: relatively naive evaluation stakeholders, the people a report actually goes to. They share a little ground that a random juror would not. Evaluators may agree a little about what counts as evidence, and perhaps about some forms of causal inference; an analysis could rely on that much without rebuilding it each time. How much they agree, and about what, is an open question, and we would rather err towards assuming too little. So we read the test strictly. We assume the reader brings ordinary sense and very little else. Whatever shared ground there is, the analysis should not depend on it.

First, the central step in evaluation is a judgement. To say "this passage supports this causal link" is to read the passage; it does not involve counting or joining. Juries make that kind of call every day, openly and as a judgement. The analyst has two options. The judgement can itself be an allowed step, declared as one, with its quotation attached and open to disagreement. Or the breaking-down can stop there, and the reader makes the judgement in the open. We take the first route; in Rubicon the judgement step is the most important component. The danger is a step that looks mechanical, such as an aggregate support score, with a contestable judgement hidden inside the number. That is worse than showing no working at all, because the judgement then looks like arithmetic.

Second, the test rules out some methods. A full Bayesian contribution analysis, for example, is beyond a lay reader. Holding to the jury test means either leaving such methods out or forcing them into steps that lose something on the way. We give up some analytic power to keep every step checkable. For reconstructible reporting we think that is the right exchange, though it is a real limit.

The steps are generic on purpose: anyone could list count, compare, group and total. A step qualifies only if a reader can follow it. The hard part, and the part particular to evaluation, is knowing where the judgement is in a given analysis, what it should be compared against, and what a reader has to be shown before reaching a verdict on a causal claim. That knowledge is what the steps are for. It is not itself one of them.

This page was drafted with the assistance of Claude, an AI model from Anthropic; the arguments and judgements are our own.